Skip to content

Pin the lazy-grammar pause and the Seed-OSS call-name constraint in tests - #1963

Merged
leehack merged 1 commit into
mainfrom
lane-1950-tbtests
Nov 12, 2026
Merged

leehack merged 1 commit into
mainfrom
lane-1950-tbtests

Conversation

@leehack

@leehack leehack commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Tracks #1950; does not close it.

Production-readiness scope

  • User-facing scope: None; test-only.
  • Supported platforms/paths: Dart VM (stories15M integration test runs on every OS; the unit test on Linux).
  • Unsupported or intentionally unavailable paths: N/A.
  • Out of scope / follow-ups: None.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly: N/A, no runtime change.
  • Public API docs, README/website docs, examples, support matrices, and changelog: N/A, test-only.
  • Regression coverage covers the audit gaps below.
  • Validation coverage registry: N/A, no capability change.
  • Security/privacy review: no secrets or paths added.
  • Follow-up work: None.

PR type guidance

  • Bugfix PR: N/A, test-only; closes two audit test gaps.

Mutation results

Gap Mutation Before restore Restored
Lazy grammar pause (test/integration/thinking_budget_grammar_test.dart) drop useLazyGrammar || in llama_cpp_service.dart new test FAILS (output ends </think>ok for both roots) passes
Seed-OSS call name (seed_oss_handler_test.dart) tool-name ::= identifier in xml_tool_call_format.dart updated test FAILS (Expected: false, Actual: true) passes

The old assertion lacked the closing </seed:tool_call>, so it was rejected as incomplete under either mutation. The new input is a complete scope with the right location parameter and a wrong function name (get_time).

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed . (touched files)
  • dart analyze
  • dart test -p vm -j 1 --exclude-tags local-only : touched files only locally (TMPDIR isolated); full lane in CI
  • dart test -p chrome --exclude-tags local-only N/A, VM-only tests
  • coverage-lib N/A, no lib/ change
  • Other: dart tool/ci/test_shards.dart hosts on both files (integration: every OS; unit: Linux only)

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
static-format-analyze format, analyze macOS PASS clean
root-vm touched tests macOS, stories15M PASS mutation table above
structured-output-adversarial compiled-grammar rejection, lazy-grammar pause macOS, stories15M PASS mutation table above
root-chrome, coverage-lib N/A N/A N/A VM-only tests, no lib/ change
high-risk-exact-head-independent-qa fresh Opus audit at 2b51d2f5 macOS, stories15M PASS M1 and M2 killed at head, survive at base; evaluator exit 2

Review Notes

High-risk regression review

  • Classification: high-risk (surface structuredOutput, from test paths only; no production file changes)
  • Implementation task: lane-1950-tbtests
  • Independent blocking QA task: fresh auditor, PASS, 0 P1/P2; evidence exit 2 (externalPrerequisitesUnavailable)
  • Exact head / current base: 2b51d2f / ec62e97
  • Production-branch deletion, bypass, or miswire proof: the mutation table; both tests fail when the guarded production line is removed
  • Affected-family real-model/artifact evidence: stories15M (lazy pause); Seed-OSS family covered by grammar recognizer
  • Explicit unavailable-family or other N/A evidence: N/A
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

@leehack
leehack marked this pull request as draft November 12, 2026 18:00
@leehack leehack mentioned this pull request Nov 12, 2026
17 of 20 tasks
@leehack
leehack marked this pull request as ready for review November 12, 2026 18:16
@leehack
leehack merged commit 3f6ccd3 into main Nov 12, 2026
42 checks passed
@leehack
leehack deleted the lane-1950-tbtests branch November 12, 2026 18:16
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1963.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant